System and method for modifying driving behavior of an autonomous vehicle based on passenger satisfaction

ABSTRACT

A system for modifying driving behavior of an autonomous vehicle based on passenger satisfaction of a passenger may include a first sensor structured to detect a first property of the passenger; a processor operably coupled to the first sensor, the processor structured to calculate a passenger satisfaction index based on the first property of the passenger; an automated driving system structured to operate the autonomous vehicle, the automated driving system being operably coupled to the processor. The processor may be structured to control the automated driving system to modify driving behavior of the autonomous vehicle in response to the passenger satisfaction index satisfying a first condition.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of and claims priority to U.S.application Ser. No. 16/389,330, filed on Apr. 19, 2019, the contents ofwhich are incorporated herein by reference.

INTRODUCTION

The subject disclosure relates to systems and methods for measuringpassenger satisfaction in an autonomous vehicle, increasing passengersatisfaction in an autonomous vehicle, and adjusting driving behavior ofan autonomous vehicle based on passenger satisfaction.

Developments in autonomous vehicle technology may allow greater accessto autonomous vehicles by the public. However, as with many newtechnologies, there may initially be a lack of trust by passengers inthe driving capabilities of autonomous vehicles. Additionally,passengers may become frustrated by the driving operation of anautonomous vehicle, especially if the passenger is unaware of the datawith which the autonomous vehicle is making decisions. These issues oflack of trust, lack of satisfaction, and passenger frustration may delayadoption of autonomous vehicle technology by the public. Additionally,present methods of studying passenger satisfaction in autonomousvehicles may rely on subjective assessment of passenger satisfactionafter the ride is completed, which may not accurately assess passengersatisfaction and may result in delays in implementing improvements tothe driving operation of the autonomous vehicle.

Accordingly, it may be desirable to provide a system and method that canaccurately assess passenger satisfaction in an autonomous vehicle inreal time as the passenger's reactions occur by detecting objectivemeasures of the passenger's behavior. Further, it may be desirable toprovide a system and method that can automatically modify drivingbehavior of the autonomous vehicle in response to the real-timeassessment of passenger satisfaction. Additionally, it may be desirableto provide a method and system for improving passenger satisfactionthrough modification of driving behavior and presentation of relevantinformation in a format that is easily understandable by passengers.

SUMMARY

In an exemplary embodiment, a system for modifying driving behavior ofan autonomous vehicle based on passenger satisfaction of a passenger mayinclude a first sensor structured to detect a first property of thepassenger. The system may further include a processor operably coupledto the first sensor, the processor structured to calculate a passengersatisfaction index based on the first property of the passenger. Thesystem may further include an automated driving system structured tooperate the autonomous vehicle, the automated driving system beingoperably coupled to the processor. The processor may be structured tocontrol the automated driving system to modify driving behavior of theautonomous vehicle in response to the passenger satisfaction indexsatisfying a first condition.

In another exemplary embodiment of the system, the first condition mayinclude a predetermined level of passenger dissatisfaction. Theprocessor may be structured to, in response to the passengersatisfaction index satisfying the predetermined level of passengerdissatisfaction, control the automated driving system to reduce amagnitude of acceleration of the autonomous vehicle, control theautomated driving system to increase a distance between the autonomousvehicle and proximate objects, or control the automated driving systemto decrease a speed of the autonomous vehicle in response to a distancebetween the autonomous vehicle and an external object being less than apredetermined threshold.

In another exemplary embodiment of the system, the first condition mayinclude a predetermined level of passenger dissatisfaction. Theprocessor may be structured to, in response to the passengersatisfaction index satisfying the predetermined level of passengerdissatisfaction, control the automated driving system to increasedeliberateness of the driving behavior of the autonomous vehicle.

In another exemplary embodiment of the system, the first property mayinclude a passenger frustration index I_(F) or a passenger trust indexI_(T).

In another exemplary embodiment of the system, the passenger frustrationindex IF may be a function of a road monitoring duration value, asecondary activity duration value, a multi-task activity transactionvalue, a side window glance value, or a facial gesture value.

In another exemplary embodiment of the system, the passenger trust indexI_(T) may be given by:

I _(T) =W ₃(DMR)+W ₄(DSA)+W ₅(MAT)+W ₆(GSW)+W ₇(FGV)

wherein DMR is the road monitoring duration value, DSA is the secondaryactivity duration value, MAT is the multi-task activity transactionvalue, GSW is the side window glance value, FGV is the facial gesturevalue, and W₃, W₄, W₅, W₆, and W₇ are weighting functions for scalingand normalization.

In another exemplary embodiment of the system, the first sensor may be acamera. The road monitoring duration value DMR may be given by:

${{DMR} = {\Sigma_{i = 0}^{n}\frac{{{EGR}\left( {x(t)} \right)}_{i}\Delta \; t}{t}}},$

wherein x(t) is a video time series output by the camera, EGR( ) is afunction that outputs a first value if the passenger is glancing at theroad, Δt is a duration of the eye glance to the road, and t is aduration of the video time series.

In another exemplary embodiment of the system, the first sensor may be acamera. The secondary activity duration value DSA may be given by:

${DSA} = {\sum_{i = 0}^{n}\frac{EGP{D\left( {x(t)} \right)}_{i}\Delta t}{t}}$

wherein x(t) is a video time series output by the camera, EGPD( ) is afunction that outputs a first value if the passenger is performing asecondary activity, Δt is a duration of performing the secondaryactivity, and t is a duration of the video time series.

In another exemplary embodiment of the system, the first sensor may be acamera. The multi-task activity transaction value MAT may be given by:

MAT=Σ _(i=0) ^(n)(EGR(x(t))_(i) +EGPD(x(t))_(i) +EGVD(x(t))_(i)+EGSW(x(t))_(i))

wherein x(t) is a video time series output by the camera, EGR( ) is afunction that outputs a first value if the passenger is glancing at theroad, EGPD( ) is a function that outputs a second value if the passengeris performing a secondary activity, EGVD( ) is a function that outputs athird value if the passenger is glancing at vehicular devices, and EGSW() is a function that outputs a fourth value if the passenger is glancingto side windows.

In another exemplary embodiment of the system, the first sensor may be acamera. The side window glance value GSW may be given by:

GSW=Σ _(i=0) ^(n)(EGSW(x(t))_(i))

wherein x(t) is a video time series output by the camera, and EGSW( ) isa function that outputs a first value if the passenger is glancing toside windows.

In another exemplary embodiment of the system, the first sensor may be acamera. The facial gesture value FGV may be given by:

FGV=Σ _(i=0) ^(n) V(FAC(x(t))_(i))

wherein x(t) is a video time series output by the camera, FAC( ) is afacial expression of the passenger, and V( ) is a function that outputsa first value if the facial expression is one of a first group of facialexpressions, and outputs a second value if the facial expression is oneof a second group of facial expressions.

In another exemplary embodiment of the system, the passenger frustrationindex IF may be a function of a galvanic skin response value, a skintemperature value, a verbal valence value, or a facial gesture value.

In another exemplary embodiment of the system, the passenger frustrationindex IF may be given by:

I _(F) =W ₈(GSR)+W ₉(ST)W ₁₀(VV)+W ₁₁(FGV)

wherein GSR is the galvanic skin response value, ST is the skintemperature value, VV is the verbal valence value, FGV is the facialgesture value, and W₈, W₉, W₁₀, W₁₁ are weighting functions for scalingand normalization.

In another exemplary embodiment of the system, the first sensor may be askin response sensor structured to output a signal indicating a galvaniccondition of the passenger's skin. The galvanic skin response GSR may begiven by:

GSR=Σ _(i=0) ^(n) F(x(t))_(i)

wherein x(t) is a signal time series of the signal output by thegalvanic skin sensor, and F( ) is a function that outputs a first valueif a signal level satisfies a first predetermined criteria.

In another exemplary embodiment of the system, the first sensor may be atemperature sensor structured to output a signal that indicates atemperature of the passenger's skin. The skin temperature value ST maybe given by:

ST=Σ _(i=0) ^(n) F(x(t))_(i)

wherein x(t) is a signal time series of the signal output by thetemperature sensor, and F( ) is a function that outputs a first value ifa signal level satisfies a first predetermined criteria.

In another exemplary embodiment of the system, the first sensor may be amicrophone. The verbal valence value may be given by:

VV=Σ _(i=0) ^(n) S(Verbal(x(t)))_(i)

wherein x(t) is a sound time series of the output of the microphone,Verbal( ) is a word spoken by the passenger, and S( ) is a function thatoutputs a first value if the word spoken by the passenger is one of afirst group of words, and outputs a second value if the word spoken bythe passenger is one of a second group of words.

In another exemplary embodiment of the system, the first sensor may be acamera. The facial gesture value FGV may be given by:

FGV=Σ _(i=0) ^(n) V(FAC(x(t))_(i))

wherein x(t) is a video time series output by the camera, FAC( ) is afacial expression of the passenger, and V( ) is a function that outputsa first value if the facial expression is one of a first group of facialexpressions, and outputs a second value if the facial expression is oneof a second group of facial expressions.

In another exemplary embodiment of the system, the system may include acommunication node structured to communicate with an external device.The communication node may be operably coupled to the processor. Thefirst sensor may be a smart device worn by the passenger. Thecommunication node may be structured to communicate with the smartdevice to receive the first property. The first property may include apassenger galvanic skin response, a passenger skin temperature, or apassenger heart rate.

In another exemplary embodiment of the system, the communication nodemay be structured to receive traffic data, weather data, passengersocial data, passenger calendar data, or destination data from theexternal device. The processor may be structured to modify the passengersatisfaction index based on the traffic data, the weather data, thepassenger social data, the passenger calendar data, or the destinationdata.

In an exemplary embodiment, a method for modifying driving behavior ofan autonomous vehicle based on passenger satisfaction of a passenger mayinclude providing an autonomous vehicle comprising a first sensor, aprocessor, and an automated driving system structured to operate theautonomous vehicle, the first sensor, the processor and the automateddriving system being operably coupled. The method may further includedetecting, with a first sensor, a first property of the passenger. Themethod may further include calculating, with a processor, a passengersatisfaction index based on the first property of the passenger. Themethod may further include controlling the automated driving system tomodify driving behavior of the autonomous vehicle in response to thepassenger satisfaction index satisfying a first condition.

The above features and advantages, and other features and advantages ofthe disclosure are readily apparent from the following detaileddescription when taken in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features, advantages and details appear, by way of example only,in the following detailed description, the detailed descriptionreferring to the drawings in which:

FIG. 1 is a schematic diagram illustrating an exemplary embodiment of asystem for measuring passenger satisfaction;

FIG. 2 is a schematic diagram illustrating an exemplary embodiment of asystem for measuring passenger satisfaction;

FIG. 3 is a schematic diagram illustrating an exemplary embodiment of asystem for measuring passenger satisfaction;

FIG. 4 is a schematic diagram illustrating an exemplary embodiment of asystem for measuring passenger satisfaction;

FIG. 5 is a schematic diagram illustrating an exemplary embodiment of asystem for measuring passenger satisfaction;

FIG. 6 is a schematic diagram illustrating an exemplary embodiment of asystem for measuring passenger satisfaction;

FIG. 7 is a schematic diagram illustrating an exemplary embodiment of asystem for measuring passenger satisfaction;

FIG. 8 is a flowchart diagram of an exemplary embodiment of a method formeasuring passenger satisfaction;

FIG. 9 is a flowchart diagram of an exemplary embodiment of a method formeasuring passenger satisfaction;

FIG. 10 is a flowchart diagram of an exemplary embodiment of a methodfor measuring passenger satisfaction;

FIG. 11 is a flowchart diagram of an exemplary embodiment of a methodfor measuring passenger satisfaction;

FIG. 12 is a flowchart diagram of an exemplary embodiment of a methodfor measuring passenger satisfaction;

FIG. 13 is a flowchart diagram of an exemplary embodiment of a methodfor measuring passenger satisfaction;

FIG. 14 is a flowchart diagram of an exemplary embodiment of a methodfor measuring passenger satisfaction;

FIG. 15 is a flowchart diagram of an exemplary embodiment of a methodfor measuring passenger satisfaction;

FIG. 16 is a flowchart diagram of an exemplary embodiment of a methodfor measuring passenger satisfaction;

FIG. 17 is a flowchart diagram of an exemplary embodiment of a methodfor measuring passenger satisfaction;

FIG. 18 is a flowchart diagram of an exemplary embodiment of a methodfor measuring passenger satisfaction;

FIG. 19 is a flowchart diagram of an exemplary embodiment of a methodfor measuring passenger satisfaction;

FIG. 20 is a schematic diagram illustrating an exemplary embodiment of asystem for modifying driving behavior of an autonomous vehicle based onpassenger satisfaction;

FIG. 21 is an exemplary embodiment illustrating a modification ofdriving behavior;

FIG. 22 is an exemplary embodiment illustrating a modification ofdriving behavior;

FIG. 23 is a schematic diagram illustrating an exemplary embodiment of asystem for modifying driving behavior of an autonomous vehicle based onpassenger satisfaction;

FIG. 24 is a flowchart diagram of an exemplary embodiment of a methodfor modifying driving behavior of an autonomous vehicle based onpassenger satisfaction;

FIG. 25 is a flowchart diagram of an exemplary embodiment of a methodfor modifying driving behavior of an autonomous vehicle based onpassenger satisfaction;

FIG. 26 is a flowchart diagram of an exemplary embodiment of a methodfor modifying driving behavior of an autonomous vehicle based onpassenger satisfaction;

FIG. 27 is a flowchart diagram of an exemplary embodiment of a methodfor modifying driving behavior of an autonomous vehicle based onpassenger satisfaction;

FIG. 28 is a flowchart diagram of an exemplary embodiment of a methodfor modifying driving behavior of an autonomous vehicle based onpassenger satisfaction;

FIG. 29 is a schematic diagram illustrating an exemplary embodiment of asystem for increasing passenger satisfaction in operation of anautonomous vehicle;

FIG. 30 is a diagram of an exemplary embodiment of a display forincreasing passenger satisfaction in operation of an autonomous vehicle;

FIG. 31 is a comparison of an exemplary embodiment of a drivingenvironment and an exemplary embodiment of a display.

FIG. 32 is a diagram of an exemplary embodiment of a display forincreasing passenger satisfaction in operation of an autonomous vehicle;

FIG. 33 is a schematic diagram illustrating an exemplary embodiment of asystem for increasing passenger satisfaction in operation of anautonomous vehicle;

FIG. 34 is a schematic diagram illustrating an exemplary embodiment of asystem for increasing passenger satisfaction in operation of anautonomous vehicle;

FIG. 35 is a diagram of an exemplary embodiment of a display forincreasing passenger satisfaction in operation of an autonomous vehicle;

FIG. 36 is a diagram of an exemplary embodiment of a display forincreasing passenger satisfaction in operation of an autonomous vehicle;

FIG. 37 is a flowchart diagram illustrating an exemplary embodiment of amethod for increasing passenger satisfaction in operation of anautonomous vehicle;

FIG. 38 is a flowchart diagram illustrating an exemplary embodiment of amethod for increasing passenger satisfaction in operation of anautonomous vehicle; and

FIG. 39 is a flowchart diagram illustrating an exemplary embodiment of amethod for increasing passenger satisfaction in operation of anautonomous vehicle.

DETAILED DESCRIPTION

The following description is merely exemplary in nature and is notintended to limit the present disclosure, its application or uses. Itshould be understood that throughout the drawings, correspondingreference numerals indicate like or corresponding parts and features. Asused herein, the term module refers to processing circuitry that mayinclude an application specific integrated circuit (ASIC), an electroniccircuit, a processor (shared, dedicated, or group) and memory thatexecutes one or more software or firmware programs, a combinationallogic circuit, and/or other suitable components that provide thedescribed functionality.

In accordance with an exemplary embodiment, FIGS. 1-2 show a system 12for measuring passenger satisfaction of a passenger 14 in a vehicle 10.Vehicle 10 may be an autonomous vehicle. System 12 may include a firstsensor 16 and a processor 18. First sensor 16 and processor 18 may beoperably connected, such as through a bus 20. First sensor 16 andprocessor 18 may also be operably connected via wireless communication.First sensor 16 may be structured to detect a first property ofpassenger 14. Exemplary embodiments of the first sensor 16 and the firstproperty will be described in detail below. Processor 18 may bestructured to calculate a passenger satisfaction index of passenger 14based on the first property of passenger 14 detected by sensor 16.

Additionally, FIG. 1 shows that in at least an embodiment, system 12 mayfurther include second sensor 22. Second sensor 22 may be operablycoupled to processor 16, and second sensor 22 may be structured todetect a second property of passenger 14. The second property may bedifferent than the first property. In an embodiment in which secondsensor 22 is present, processor 18 may be structured to calculate thepassenger satisfaction index of passenger 14 based on the first propertyand the second property. The first property and the second property maybe objective measurements of a response of passenger 14 to operation ofthe autonomous vehicle, and may include biometric data of passenger 14,interpretation of gestures or words of passenger 14, or a quantificationof frustration or trust of passenger 14 in the operation of autonomousvehicle 10. The first property of passenger 14 is discussed in greaterdetail below with illustrative examples.

The first property may include a quantification of the frustration levelof passenger 14 of the operation of the autonomous operation of vehicle10, such as a passenger frustration index IF. Alternatively, the firstproperty may include a quantification of the trust that passenger 14 hasin the autonomous operation of vehicle 10, such as a passenger trustindex I_(T). The first property may also be a combination of thepassenger frustration index IF and passenger trust index I_(T).

In an exemplary embodiment, the passenger satisfaction index may beexpressed by the following equation:

PSI=100−W ₁(I _(T))−W ₂(I _(F))  (Eq. 1)

PSI is the passenger satisfaction index and W₁, W₂ are weightingfunctions to provide scaling of passenger trust index I_(T) andpassenger frustration index IF and restrict a sum of I_(T) and I_(F) to100 or less. As explained in detail below, passenger trust index I_(T)and passenger frustration index I_(F) can be calculated based on avariety of different types of inputs. Thus, weighting functions W₁ andW₂ can be used to normalize and weight the passenger trust index I_(T)and passenger frustration index I_(F) into a unitless format for easieranalysis and comparison. Weighting functions W₁ and W₂ may be a constantcoefficient, or may be variable functions based on the type or amount ofdata being considered. Alternatively, one of the weighting functions W₁,W₂ may be set to 0 if a particular value is not being used in thecalculation of the passenger satisfaction index PSI.

In this exemplary embodiment, the passenger satisfaction index PSI isdesigned to have a baseline value of 100. However, it will be understoodthat this is not required, and the baseline value may be set as desiredby the manufacturer. For example, a baseline value of 0 could be usedinstead.

Additionally, it will be understood that the calculation of thepassenger satisfaction index PSI is not limited to subtractionoperations as shown above. The exact operations used will depend on thedefinition of the passenger trust index I_(T) and passenger frustrationindex I_(F), and how these values are calculated. For example, in oneembodiment, a higher value of passenger satisfaction index PSI mayindicate greater level of satisfaction of passenger 14, while a lowervalue of passenger satisfaction index PSI may indicate a lower level ofsatisfaction of passenger 14. Alternatively, in another embodiment, ahigher level of passenger satisfaction index PSI may indicate a loverlevel of satisfaction of passenger 14, while a lower value of passengersatisfaction index PSI may include a higher level of satisfaction ofpassenger 14. Accordingly, the sign of the operations and the sign ofthe values of passenger trust index I_(T) and passenger frustrationindex I_(F) will depend on the frame of reference for evaluatingpassenger satisfaction index PSI.

Passenger trust index I_(T) may be based on a variety of inputs based onobservation of passenger 14. For example, passenger trust index I_(T)may be calculated based on a road monitoring duration value DMR, asecondary activity duration value DSA, a multi-task activity transactionvalue MAT, a side window glance value GSW, or a facial gesture valueFGV. In an exemplary embodiment, passenger trust index I_(T) may bebased on the following equation:

I _(T) =W ₃(DMR)+W ₄(DSA)+W ₅(MAT)+W ₆(GSW)+W ₇(FGV)  (Eq. 2)

W₃, W₄, W₅, W₆, and W₇ are weighting functions for scaling andnormalization. Each of the road monitoring duration value DMR, thesecondary activity duration value DSA, the multi-task activitytransaction value MAT, the side window glance value GSW, and the facialgesture value FGV may have different ranges of possible values ordifferent magnitudes. Accordingly, weighting functions W₃, W₄, W₅, W₆,and W₇ scale and normalize the values so that they can be meaningfullycombined and analyzed. Weighting functions W₃, W₄, W₅, W₆, and W₇ may bea constant coefficient, or may be variable functions based on the typeor amount of data being considered. Alternatively, one of the weightingfunctions W₃, W₄, W₅, W₆, and W₇ may be set to 0 if a particular valueis not being used in the calculation of the passenger trust index I_(T).The signs of weighting functions W₃, W₄, W₅, W₆, and W₇ may also bevaried depending on how the specific values of the road monitoringduration value DMR, the secondary activity duration value DSA, themulti-task activity transaction value MAT, the side window glance valueGSW, and the facial gesture value FGV are calculated. For example, inone embodiment, a positive value of road monitoring duration value DMRmay indicate a low level of trust in the autonomous operation of vehicle10, while a positive value of secondary activity duration value DSA mayindicate a high level of trust in the autonomous operation of vehicle10. Accordingly the signs of weighting functions W₃ and W₄ may be setopposite to each other so that the passenger trust index I_(T)accurately reflects the meaning of the values used in the calculation.

FIG. 2 shows an exemplary embodiment of system 12 configured tocalculate the road monitoring duration value DMR, the secondary activityduration value DSA, the multi-task activity transaction value MAT, theside window glance value GSW, and the facial gesture value FGV. In FIG.2, first sensor 16 may be implemented as camera 23. Camera 23 may beprovided in the dashboard 24 of vehicle 10, on an interior ceiling ofvehicle 10, in a steering wheel 26, or any position where the camera canrecord the face of passenger 14. Lines of sight of cameras 23 are shownby dashed line in FIG. 2. While FIG. 2 shows multiple cameras 23, itwill be understood that this is for illustrative purposes to show atleast some possible positions of camera 23, and it will be furtherunderstood that a single camera 23 may be provided. Camera 23 may bestructured to record and output a video time series of the face and headof passenger 14. Camera 23 and/or processor 18 (shown in FIG. 1) may beused to calculate a view direction of passenger 14 based on analysis ofthe video time series recorded by camera 23.

The road monitoring duration value DMR may be calculated based on thevideo time series output by camera 23 as given by the followingequation:

$\begin{matrix}{{DMR} = {\sum_{i = 0}^{n}\frac{EG{R\left( {x(t)} \right)}_{i}\Delta t}{t}}} & \left( {{Eq}.\mspace{11mu} 3} \right)\end{matrix}$

wherein x(t) is a video time series output by the camera, EGR( ) is afunction that outputs a first value if passenger 14 is glancing at theroad, Δt is a duration of the eye glance to the road, and t is aduration of the video time series.

The function EGR( ) may return a value of 1 if passenger 14 is glancingat the road and 0 otherwise. Alternatively, function EGR( ) may bedesigned to assign more weight (i.e., a higher value) to glances thatlast for a longer time. The term Δt/t may be included as shown above inEq. 3 as a way to weight the value by a ratio of glance period to theoverall time of the video time series.

It will be understood that Eq. 3 is not the only equation possible forcalculating road monitoring duration value DMR, and that the equationmay be modified based on the specific needs of specific applications.Regardless of the specific equation used to calculate road monitoringduration value DMR, it will be understood that the calculation of roadmonitoring duration value DMR returns a value indicative of the amountof time that passenger 14 spends looking at the road. If passenger 14spends a large percentage of time looking at the road, this couldindicate a lower level of trust in the autonomous operation of vehicle10, and the passenger trust index I_(T) and passenger satisfaction indexPSI can be adjusted accordingly. Alternatively, if the passenger is notlooking at the road (e.g., looking at a phone, a book, or otherpassengers), this could indicate a higher level of trust in theautonomous operation of vehicle 10, and the passenger trust index I_(T)and passenger satisfaction index PSI can be adjusted accordingly.

The secondary activity duration value DSA may be calculated based on thevideo time series output by camera 23 as given by the followingequation:

$\begin{matrix}{{DSA} = {\sum_{i = 0}^{n}\frac{EGP{D\left( {x(t)} \right)}_{i}\Delta t}{t}}} & \left( {{Eq}.\mspace{11mu} 4} \right)\end{matrix}$

wherein x(t) is a video time series output by the camera, EGPD( ) is afunction that outputs a first value if passenger 14 performs a secondaryactivity, Δt is a duration of performing the secondary activity, and tis a duration of the video time series.

The function EGPD( ) may return a value of 1 if passenger 14 isperforming a secondary activity such as using a phone, reading, lookingat other passengers, or the like. Alternatively, function EGPD( ) may bedesigned to assign more weight (i.e., a higher value) to secondaryactivities that last for a longer time. The term Δt/t may be included asshown above in Eq. 4 as a way to weight the value by a ratio ofsecondary activity time to the overall time of the video time series.

It will be understood that Eq. 4 is not the only equation possible forcalculating secondary activity duration value DSA, and that the equationmay be modified based on the specific needs of specific applications.Regardless of the specific equation used to calculate secondary activityduration value DSA, it will be understood that the calculation ofsecondary activity duration value DSA returns a value indicative of theamount of time that passenger 14 spends doing activities unrelated tothe operation of vehicle 10. If passenger 14 spends a large percentageof time on secondary activities, this could indicate a higher level oftrust in the autonomous operation of vehicle 10, and the passenger trustindex I_(T) and passenger satisfaction index PSI can be adjustedaccordingly. Alternatively, if the passenger does not spend much time onsecondary activities, this could indicate a lower level of trust in theautonomous operation of vehicle 10, and the passenger trust index I_(T)and passenger satisfaction index PSI can be adjusted accordingly.

The secondary side window glance value GSW may be calculated based onthe video time series output by camera 23 as given by the followingequation:

GSW=Σ _(i=0) ^(n)(EGSW(x(t))_(i))  (Eq. 5)

wherein x(t) is a video time series output by the camera, and EGSW( ) isa function that outputs a first value if the passenger is glancing toside windows.

The function EGSW( ) may return a value of 1 if passenger 14 glances outa side window of vehicle 10. Alternatively, function EGSW( ) may bedesigned to assign more weight (i.e., a higher value) to longer glancesor more frequent glances out the side window.

It will be understood that Eq. 5 is not the only equation possible forcalculating side window glance value GSW, and that the equation may bemodified based on the specific needs of specific applications.Regardless of the specific equation used to calculate side window glancevalue GSW, it will be understood that the calculation of side windowglance value GSW returns a value indicative the number of times or theamount of time that passenger 14 spends looking out the side window ofvehicle 10. If passenger 14 spends a large percentage of time lookingout the side window, this could indicate a higher level of trust in theautonomous operation of vehicle 10, and the passenger trust index I_(T)and passenger satisfaction index PSI can be adjusted accordingly.Alternatively, if passenger 14 does not look out the side window veryoften, this could indicate a lower level of trust in the autonomousoperation of vehicle 10 and the passenger trust index I_(T) andpassenger satisfaction index PSI can be adjusted accordingly.

The multi-task activity transaction value MAT may be calculated based onthe video time series output by camera 23 as given by the followingequation:

MAT=Σ _(i=0) ^(n)(EGR(x(t))_(i) +EGPD(x(t))_(i) +EGVD(x(t))_(i)+EGSW(x(t))_(i))  (Eq. 6)

wherein x(t) is a video time series output by the camera, EGR( ) is afunction that outputs a first value if passenger 14 is glancing at theroad, EGPD( ) is a function that outputs a second value if passenger 14is performing a secondary activity; EGVD( ) is a function that outputs athird value if passenger 14 is glancing at vehicular devices; and EGSW() is a function that outputs a fourth value if passenger 14 is glancingto side windows.

Functions EGR( ), EGPD( ), and EGSW( ) are described in detail above.Function EGVD( ) may return a value of 1 if passenger 14 glances atvehicular devices such as controls, gauges, or displays. AlternativelyEGVD( ) may be designed to adjust the value depending on the length ofthe glance at the vehicular device.

It will be understood that Eq. 6 is not the only equation possible formulti-task activity transaction value MAT, and that the equation may bemodified based on the specific needs of specific applications.Regardless of the specific equation used to calculate multi-taskactivity transaction value MAT, it will be understood the calculation ofmulti-task activity transaction value MAT returns a value indicative ofa level of multi-tasking by passenger 14. If passenger 14 spends a largeamount of time multi-tasking, this could indicate a higher level oftrust in the autonomous operation of vehicle 10, and the passenger trustindex I_(T) and passenger satisfaction index PSI can be adjustedaccordingly. Alternatively, if passenger 14 does not spend much timemulti-tasking, this could indicate a lower level of trust in theautonomous operation of vehicle 10, and the passenger trust index I_(T)and passenger satisfaction index PSI can be adjusted accordingly. Evenif the passenger 14 does not spend much time multi-tasking, thecalculation of multi-task activity transaction value MAT could bemodified to still reflect a high level of trust in autonomous operationof vehicle 10 if the majority of attention of passenger 14 is focused onsecondary activities or looking out the side windows, for example.

The facial gesture value FGV may be calculated based on the video timeseries output by camera 23 as given by the following equation:

FGV=Σ _(i=0) ^(n) V(FAC(x(t))_(i))  (Eq. 7)

wherein x(t) is the video time series output by the camera, FAC( ) is afacial expression of passenger 14; and V( ) is a function that outputs afirst value if the facial expression is one of a first group of facialexpressions, and outputs a second value if the facial expression is oneof a second group of facial expressions.

Camera 23 and processor 18 (shown in FIG. 1) may be structured tointerpret a face of passenger 14 and classify a facial expression ofpassenger 14 into one of two groups. This classification can be based onrecognizing patterns in the shape and structure of eyebrows, eyes,mouth, and other relevant features of the face of passenger 14. Forexample, the first group may include positive facial expressions such assurprise, happiness, interest, or calm. The second group may includenegative facial expressions such as sadness, fear, anger, disgust,contempt, horror, discomfort, or stoicism. If the function FAC( )determines that a facial gesture of passenger 14 is in the first group,the function V( ) could increment the facial gesture value FGV by 1.Alternatively, if the function FAC( ) determines that a facial gestureof passenger 14 is in the second group, the function V( ) coulddecrement the facial gesture value FGV by 1. In this way, facial gesturevalue FGV could be a net measure of the trust of passenger 14 in theoperation of autonomous vehicle 10. In other words, a facial gesturevalue FGV indicating more positive facial gestures could indicate ahigher level of trust in the autonomous operation of vehicle 10, and thepassenger trust index I_(T) and passenger satisfaction index PSI can beadjusted accordingly. Alternatively, a facial gesture value FGVindicating more negative facial gestures could indicate a lower level oftrust in the autonomous operation of vehicle 10, and the passenger trustindex I_(T) and passenger satisfaction index PSI can be adjustedaccordingly.

Passenger frustration index I_(F) may be based on a variety of inputsbased on observation of passenger 14. For example, passenger frustrationindex I_(F) may be calculated based on a galvanic skin response valueGSR, a skin temperature value ST, a verbal valence value VV, or a facialgesture value FGV. In an exemplary embodiment, passenger frustrationindex I_(F) may be based on the following equation:

I _(F) =W ₈(GSR)+W ₉(ST)W ₁₀(VV)+W ₁₁(FGV)  (Eq. 8)

W₈, W₉, W₁₀, and W₁₁ are weighting functions for scaling andnormalization. Each of the galvanic skin response value GSR, the skintemperature value ST, the verbal valence value VV, and the facialgesture value FGV may have different ranges of possible values ordifferent magnitudes. Accordingly, weighting functions W₈, W₉, W₁₀, andW₁₁ scale and normalize the values so that they can be meaningfullycombined and analyzed. Weighting functions W₈, W₉, W₁₀, and W₁₁ may be aconstant coefficient, or may be variable functions based on the type oramount of data being considered. Alternatively, one of the weightingfunctions W₈, W₉, W₁₀, and W₁₁ may be set to 0 if a particular value isnot being used in the calculation of the passenger frustration indexI_(T). Additionally, the signs of weighting functions W₈, W₉, W₁₀, andW₁₁ may also be varied depending on how the specific values of thegalvanic skin response value GSR, the skin temperature value ST, theverbal valence value VV, and the facial gesture value FGV arecalculated. For example, in one embodiment, a positive value of galvanicskin response value GSR may indicate a high level of frustration withthe autonomous operation of vehicle 10, while a positive value of facialgesture value FGV may indicate a low level of frustration with theautonomous operation of vehicle 10. Accordingly, the signs of weightingfunctions W₈ and W₁₁ may be set opposite to each other so that thepassenger frustration index I_(F) accurately reflects the meaning of thevalues used in the calculation.

FIG. 3 shows an exemplary embodiment of system 12 configured tocalculate the galvanic skin response value GSR or the skin temperaturevalue ST. In FIG. 3, first sensor 16 may be implemented as a skin sensor30. Skin sensor 30 may be provided in steering wheel 26, and may bestructured to detect electrical properties of skin of passenger 14(e.g., resistance, voltage, current, capacitance, etc.) or detect a skintemperature of passenger 14. Alternatively, it may be possible toprovide a skin sensor 30 within seat 28 if calibrated properly toaccount for clothing and set material between the skin sensor 30 and theskin of passenger 14. While FIG. 3 shows multiple skin sensors 30, itwill be understood that this is for illustrative purposes to show atleast some possible positions of skin sensor 30, and it will be furtherunderstood that a single skin sensor 30 may be provided. Skin sensor 30is structured to output a signal corresponding to the electricalproperties of the skin of passenger 14 or the temperature of the skin ofpassenger 14, and the signal can be analyzed and processed by processor18 (see FIG. 1).

The galvanic skin response value may be calculated based on the signalfrom skin sensor 30 configured to detect electrical properties as givenby the following equation:

GSR=Σ _(i=0) ^(n) F(x(t))_(i)  (Eq. 9)

wherein x(t) is a signal time series of the signal output by skin sensor30; and F( ) is a function that outputs a first value if a signal levelsatisfies a first predetermined criteria.

For example, the function F( ) may increment the galvanic skin responseby a set value if electrical properties of the skin of passenger 14exceed or are lower than a specified threshold. Alternatively, functionF( ) may increment the galvanic skin response GSR by a set value ifelectrical properties of the skin of passenger 14 change by a specifiedamount or for a specified duration of time.

It will be understood that Eq. 9 is not the only equation possible forcalculating galvanic skin response GSR, and that the equation may bemodified based on the specific needs of specific applications.Regardless of the specific equation used to calculate galvanic skinresponse value GSR, it will be understood that the calculation ofgalvanic skin direction value GSR returns a value indicative of anelectrical response of passenger 14. High levels of electrical responseor rapid or large changes in electrical response of passenger 14 couldindicate a higher level of frustration with the autonomous operation ofvehicle 10, and the passenger frustration index I_(F) and passengersatisfaction index PSI could be adjusted accordingly. Alternatively, lowlevels of electrical response or a relatively constant level ofelectrical response of passenger 14 could indicate a lower level offrustration with the autonomous operation of vehicle 10, and thepassenger frustration index I_(F) and passenger satisfaction index PSIcould be adjusted accordingly.

The skin temperature value ST may be calculated based on signals from askin sensor 30 configured to detect temperature. Alternatively, skintemperature value ST may be calculated from images captured by camera 23(see FIG. 2) if camera 23 includes infrared elements structured todetect temperature. Skin temperature value ST may be given by thefollowing equation:

ST=Σ _(i=0) ^(n) F(x(t))_(i)  (Eq. 10)

wherein x(t) is a signal time series of the signal output by skin sensor30 or camera 23; and F( ) is a function that outputs a first value if asignal level satisfies a first predetermined criteria.

For example, function F( ) may increment the skin temperature value STby a set value if the skin temperature of passenger 14 exceeds or islower than a specified threshold. Alternatively, function F( ) mayincrement the skin temperature value ST by a set value if the skintemperature of passenger 14 changes by a specified amount or for aspecified duration of time.

It will be understood that Eq. 10 is not the only equation possible forcalculating skin temperature response ST, and that the equation may bemodified based on the specific needs of specific applications.Regardless of the specific equation used to calculate skin temperatureresponse ST, it will be understood that the calculation of skintemperature response ST returns a value indicative the skin temperatureof passenger 14. High levels of skin temperature or rapid or largechanges in skin temperature of passenger 14 could indicate a higherlevel of frustration with the autonomous operation of vehicle 10, andthe passenger frustration index I_(F) and passenger satisfaction indexPSI could be adjusted accordingly. Alternatively, low levels of skintemperature or relatively constant level of skin temperature couldindicate a lower level of frustration with the autonomous operation ofvehicle 10, and the passenger frustration index I_(F) and passengersatisfaction index PSI could be adjusted accordingly.

FIG. 4 shows an exemplary embodiment of system 12 configured tocalculate verbal valence value VV. In FIG. 3 first sensor 16 may beimplemented as a microphone 32. Microphone 32 may be provided indashboard 24, provided in steering wheel 26, provided on an interiorceiling of vehicle 10, or any other suitable position where the voice ofpassenger 14 can be detected. While FIG. 4 shows multiple microphones32, it will be understood this this is for illustrative purposes to showat least some possible positions of microphone 32, and it will befurther understood that a single microphone 32 may be provided.Microphone 32 may be structured to record and output a sound time seriesof an audio environment inside vehicle 10.

The verbal valence value may be calculated based on the sound timeseries recorded by microphone 32 as given by the following equation:

VV=Σ _(i=0) ^(n) S(Verbal(x(t)))_(i)  (Eq. 11)

wherein x(t) is a sound time series of the output of microphone 32;Verbal( ) is function identifying a word spoken by passenger 14; and S() is a function that outputs a first value if the word spoken bypassenger 14 is one of a first group of words, and outputs a secondvalue if the word spoken by passenger 14 is one of a second group ofwords.

Microphone 32 and processor 18 (see FIG. 1) may be structured toidentify and interpret a word or phrase spoken by passenger 14. It willbe understood that in the description below, references to a word spokenby passenger 14 will also include a phrase spoken by passenger 14. Theword spoken by passenger 14 can be classified into one of two groups.For example, the first group may include positive words such as “nice,”“good,” “best,” “comfortable,” “better,” “great,” “more,” “smooth,”“fantastic,” “safe,” etc. The second group may include negative wordssuch as “than I would,” “would have,” “not,” “concerned,” “whew,”“uncomfortable,” “nervous,” “don't,” “terrible,” “didn't,” “reckless,”“unsafe,” “less,” “dangerous,” “extreme,” “wow,” etc. If functionVerbal( ) determines that a word is in the first group, the function S() could increment the verbal valence value VV by 1. Alternatively, ifthe function Verbal( ) determines that a word is in the second group,the function V( ) could decrement the verbal valence value VV by 1. Inthis way, verbal valence value could be a net measure of the frustrationof user 14 with the autonomous operation of vehicle 10 based on thespoken utterances of passenger 14. In other words, a verbal valencevalue VV indicating more positive words could indicate a lower level offrustration with the autonomous operation of vehicle 10, and thepassenger frustration index I_(F) and passenger satisfaction index PSIcan be adjusted accordingly. Alternatively, a verbal valence valueindicating more negative words could indicate a higher level offrustration with the autonomous operation of vehicle 10, and thepassenger frustration index I_(F) and passenger satisfaction index PSIcan be adjusted accordingly.

With respect to passenger frustration index IF, the facial gesture valueFGV may be calculated in similar fashion as described above with respectto the passenger trust index I_(T).

FIG. 5 shows an exemplary embodiment in which system 12 includes acommunication node 34 structured to communicate with an external device36. Communication node 34 may be operably connected to processor 18 sothat processor 18 can receive and operate on data received viacommunication node 34. Communication node 34 may be any type oftransmitter/receiver capable of wireless communication. For example,communication node 34 may be a WI-FI transmitter/receiver, a Bluetoothtransmitter receiver, an RF transmitter/receiver, a cellulartransmitter/receiver, or other suitable type of device.

External device 36 may be any type of device capable of wirelesscommunication with communication node 34. FIG. 6 shows exemplaryembodiments of possible external devices that vehicle 10 couldcommunicate with via communication node 34 (see FIG. 3). For example,external device 36 may be another vehicle 38 equipped with acommunication node, a smartphone 40, a computer 42 accessible via theinternet, or smart infrastructure objects such as stoplight 44.

In these embodiments, processor 18 could receive information such aslocal traffic data, local weather data, passenger social data, passengercalendar data, or destination data from an external device 36. Thisinformation could be used by processor 18 to modify the passengersatisfaction index PSI. For example, if there is heavy traffic or badweather, processor 18 could modify the passenger satisfaction index PSIto reflect that passenger 14 may be feeling additional nervousness inthese situations. Alternatively, with access to social data, calendardata, or destination, processor 18 could modify the passengersatisfaction index to reflect where the passenger is headed (e.g.,happier for social gathering, more nervous for a client meeting, etc.),and processor 18 could also modify the autonomous operation of vehicle18 to account for important events, such as making it to an airport intime for a flight. For example, passenger 14 may exhibit objectivecriteria indicating a high level of frustration, i.e., a high passengerfrustration index I_(F), but if processor 18 is aware that passenger ispossibly late for an appointment or a flight, this could be taken intoaccount and autonomous operation of vehicle 10 could be adjustedaccordingly. Adjusting autonomous operation of vehicle 10 will bedescribed in greater detail below.

FIG. 7 shows additional exemplary embodiments of external device 36implemented as wearable smart devices worn by passenger 14. For example,external device 36 may be a smart watch 46, smart glasses 48, or abiometric sensor 49 worn by the passenger such as a heart monitor, bloodpressure monitor, glucose monitor, etc. It will be understood that thesewearable smart devices could also serve as first sensor 16 (see FIG. 1)and transmit information such as electrical properties of skin, skintemperature, or heart rate to processor 18 via communication node 34 toaid in calculation of passenger frustration index I_(F) or passengertrust index I_(T).

FIG. 8 shows an exemplary embodiment of a method 1000 for measuringpassenger satisfaction of passenger 14. In block 102, a system 12 isprovided including first sensor 16 and processor 18. System 12, firstsensor 16, and processor 18 are described in detail herein. In block104, first sensor 16 is used to detect a first property of passenger 14.In block 106, processor 18 calculates passenger satisfaction index PSIbased on the first property received from first sensor 16. Exemplaryembodiments of detecting a first property of passenger 14 andcalculating passenger satisfaction index PSI are described herein.

FIG. 9 shows an exemplary embodiment of a method 1100 for measuringpassenger satisfaction of passenger 14. In block 110, a system 12 (seeFIG. 1) is provided including first sensor 16, processor 18, and secondsensor 22. In block 112, first sensor 16 is used to detect a firstproperty of passenger 14. In block 114, second sensor 14 is used todetect a second property of passenger 14. In block 116, processor 18calculates passenger satisfaction PSI based on the first propertyreceived from first sensor 16 and the second property received fromsecond sensor 18.

FIG. 10 shows an exemplary embodiment of a method 1200 for measuringpassenger satisfaction of passenger 14. In block 120, a system 12 (seeFIGS. 1-2) is provided including processor 18 and camera 23. In block122, camera 23 is used to record a video time series. In block 124,processor 18 calculates a road monitoring duration value DMR based onthe video time series, as described in detail above. In block 126,processor 18 calculates passenger satisfaction index PSI based on roadmonitoring duration value DMR.

FIG. 11 shows an exemplary embodiment of a method 1300 for measuringpassenger satisfaction of passenger 14. In block 120, a system 12 (seeFIGS. 1-2) is provided including processor 18 and camera 23. In block122, camera 23 is used to record a video time series. In block 134,processor 18 calculates a secondary activity duration value DSA based onthe video time series, as described in detail above. In block 136,processor 18 calculates passenger satisfaction index PSI based onsecondary activity duration value DSA.

FIG. 12 shows an exemplary embodiment of a method 1400 for measuringpassenger satisfaction of passenger 14. In block 120, a system 12 (seeFIGS. 1-2) is provided including processor 18 and camera 23. In block122, camera 23 is used to record a video time series. In block 144,processor 18 calculates a multi-task activity transaction value MATbased on the video time series, as described in detail above. In block146, processor 18 calculates passenger satisfaction index PSI based onmulti-task activity transaction value MAT.

FIG. 13 shows an exemplary embodiment of a method 1500 for measuringpassenger satisfaction of passenger 14. In block 120, a system 12 (seeFIGS. 1-2) is provided including processor 18 and camera 23. In block122, camera 23 is used to record a video time series. In block 154,processor 18 calculates a side window glance value GSW based on thevideo time series, as described in detail above. In block 156, processor18 calculates passenger satisfaction index PSI based on side windowglance value GSW.

FIG. 14 shows an exemplary embodiment of a method 1600 for measuringpassenger satisfaction of passenger 14. In block 120, a system 12 (seeFIGS. 1-2) is provided including processor 18 and camera 23. In block122, camera 23 is used to record a video time series. In block 164,processor 18 calculates a facial gesture value FGV based on the videotime series, as described in detail above. In block 166, processor 18calculates passenger satisfaction index PSI based on facial gesturevalue FGV.

FIG. 15 shows an exemplary embodiment of a method 1700 for measuringpassenger satisfaction of passenger 14. In block 170, a system 12 (seeFIGS. 1 and 3) is provided including processor 18 and skin sensor 30. Inblock 172, skin sensor 30 is used to detect a galvanic condition of theskin of passenger 14. In block 174, processor 18 calculates galvanicskin response value GSR based on the galvanic condition detected by skinsensor 30, as described in detail above. In block 176, processor 18calculates passenger satisfaction index PSI based on galvanic skinresponse value GSR.

FIG. 16 shows an exemplary embodiment of a method 1800 for measuringpassenger satisfaction of passenger 14. In block 180, a system 12 (seeFIGS. 1 and 3) is provided including processor 18 and skin sensor 30. Inblock 182, skin sensor 30 is used to detect a skin temperature ofpassenger 14. In block 184, processor 18 calculates skin temperaturevalue ST based on the skin temperature detected by skin sensor 30, asdescribed in detail above. In block 186, processor 18 calculatespassenger satisfaction index PSI based on skin temperature value ST.

FIG. 17 shows an exemplary embodiment of a method 1900 for measuringpassenger satisfaction of passenger 14. In block 190, a system 12 (seeFIGS. 1 and 4) is provided including processor 18 and microphone 32. Inblock 192, microphone 32 is used to record a sound time series. In block194, processor 18 calculates verbal valence value VV based on the soundtime series recorded by microphone 32, as described in detail above. Inblock 196, processor 18 calculates passenger satisfaction index PSIbased on verbal valence value VV.

FIG. 18 shows an exemplary embodiment of a method 2000 for measuringpassenger satisfaction of passenger 14. In block 200, a system 12 (seeFIGS. 5-7) is provided including processor 18, a communication node 34,and a wearable smart device as first sensor 16. In block 202, thewearable smart device is used to measure biometric data of passenger 14such as galvanic skin response, skin temperature or heart rate. In block204, processor 18 receives the biometric data from the wearable smartdevice via communication node 34. In block 206, processor 18 calculatesthe passenger satisfaction index PSI based on the biometric datareceived from the wearable smart devices.

FIG. 19 shows an exemplary embodiment of a method 2100 for measuringpassenger satisfaction of passenger 14. In block 210, a system 12 (seeFIGS. 5-6) is provided including processor 18 and a communication node34. In block 212, communication node 34 communicates with externaldevice 36 to receive information such as traffic data, weather data,passenger social data, passenger calendar data, or destination data. Inblock 214, process 18 modifies the passenger satisfaction index PSIbased on the traffic data, weather data, passenger social data,passenger calendar data, or destination data.

In accordance with an exemplary embodiment, FIG. 20 shows a system 50for modifying driving behavior of an autonomous vehicle 10 based onpassenger satisfaction. System 50 may include first sensor 16, processor18, and automated driving system 52. First sensor 16 and processor 18may be similar to the structures described in detail herein.Additionally, processor 18 may be operably connected to automateddriving system 52 and structured to control automated driving system 52.Automated driving system 52 may include or be operably connected tovarious controllers and sensors for detecting a driving environment andcontrolling speed, acceleration, braking, and steering of the autonomousvehicle 10 based on a vehicle path plan calculated by processor 18. Thevehicle path plan may include a series of maneuvers planned forautonomous vehicle 10 based on a desired destination and the localdriving environment. In other words, automated driving system controlssystems such as braking, acceleration, and steering to operate theautonomous vehicle. Processor 18 may be further structured to controlthe automated driving system 52 to modify driving behavior in responseto the passenger satisfaction index PSI satisfying a first condition.Passenger satisfaction index PSI may be calculated as described indetail herein.

The first condition may be a predetermined level of passengerdissatisfaction. To satisfy the first condition, the calculatedpassenger satisfaction index may need to exceed a predeterminedthreshold. Processor 18 may be structured so as to, in response topassenger 14 exhibiting the predetermined level of passengerdissatisfaction, control automated driving system 52 to increasedeliberateness of the driving behavior of autonomous vehicle 10. Inother words, processor 18 controls automated driving system 52 tocontrol the driving behavior to drive in a more careful manner, so as toease the dissatisfaction of passenger 14.

Deliberateness of the driving behavior of vehicle 10 may be varied in avariety of ways. For example, a magnitude of maximum acceleration ofvehicle 10 may be reduced from a baseline value. In words, processor 18will control automated driving system 52 such that autonomous vehicle 10will accelerate and decelerate more gradually, so as to decrease anxietyand frustration of passenger 14 and increase trust in the automatedvehicle 10. Additionally, increasing deliberateness of the drivingbehavior of vehicle 10 may include reducing overall vehicle speed.

Deliberateness may also be increased by controlling automated drivingsystem 52 to maintain a greater distance between autonomous vehicle 10and nearby or proximate objects. FIG. 21 shows an exemplary embodimentof a baseline driving behavior in which autonomous vehicle 10 maintainsa predetermined following distance D1 from second vehicle 54. However,if the passenger satisfaction index PSI satisfies the predeterminedlevel of dissatisfaction, then processor 18 may control automateddriving system 52 to maintain a second following distance D2 that islarger than following distance D1, as seen in FIG. 22. The largerfollowing distance D2 may ease frustration and anxiety of passenger 14and increase trust in autonomous vehicle 10. In similar fashion,processor 18 may control automated driving system 52 to keep a largerdistance from other objects such as pedestrians or bicyclists. Processor18 may further control automated driving system 52 to decrease speedwhen a distance between autonomous vehicle 10 and a nearby object isless than a predetermined threshold. For example, if autonomous vehicle10 gets too close to a pedestrian or bicyclist, processor 18 can controlautomated driving system 18 to further reduce speed.

Overall, increasing the deliberateness of the driving behavior is meantto increase the overall carefulness, caution, and courtesy with whichautomated vehicle 10 is being operated, in order to decrease anxiety andfrustration of passenger 14 and increase trust in autonomous vehicle 10.

FIG. 23 illustrates another exemplary embodiment of the system formodifying driving behavior of an autonomous vehicle. In FIG. 23, system50 may include a display 56 operably coupled to processor 18. Processor18 may control display 56 to notify passenger 14 of an upcoming changein driving behavior of autonomous vehicle 10. For example, display 56may display a notification that vehicle speed is being changed, thatfollowing distance is being changed, that a lane change is being made,etc. In this way, display 56 informs passenger 56 that autonomousvehicle is responding to potential concerns of passenger 14, therebyfurther increasing the trust and confidence of passenger 14 inautonomous vehicle 10.

In accordance with an exemplary embodiment, FIG. 24 shows a method 2200for modifying driving behavior of an autonomous vehicle based onpassenger satisfaction of passenger 14. In block 220, a system 50 isprovided comprising first sensor 16, processor 18, and automated drivingsystem 52. In block 222, first sensor 16 is used to detect a firstproperty of passenger 14. In block 224, processor 18 calculatespassenger satisfaction index PSI based on the first property detected byfirst sensor 16. In block 226, it is determined whether the passengersatisfaction index PSI satisfies a first condition. If the passengersatisfaction index PSI does satisfy the first condition (“Yes” in block226), then the method proceeds to block 228. If the passengersatisfaction index PSI does not satisfy the first condition (“No” inblock 226), then the method returns to block 222 to continue detectingthe first property of passenger 14. In block 228, processor 18 controlsautomated driving system 52 to modify driving behavior of autonomousvehicle 10. The method may then return to block 222 to continuedetecting the first property of passenger 14.

FIG. 25 shows an exemplary embodiment of a method 2300 for modifyingdriving behavior of an autonomous vehicle based on passengersatisfaction of passenger 14. In block 220, a system 50 is providedcomprising first sensor 16, processor 18, and automated driving system52. In block 222, first sensor 16 is used to detect a first property ofpassenger 14. In block 224, processor 18 calculates passengersatisfaction index PSI based on the first property detected by firstsensor 16. In block 236, it is determined whether the passengersatisfaction index PSI meets a predetermined level of passengerdissatisfaction. If the passenger satisfaction index PSI does meet apredetermined level of passenger dissatisfaction (“Yes” in block 226),then the method proceeds to block 238. If the passenger satisfactionindex PSI does not meet the predetermined level of passengerdissatisfaction (“No” in block 226), then the method returns to block222 to continue detecting the first property of passenger 14. In block238, processor 18 controls automated driving system 52 to modify drivingbehavior of autonomous vehicle 10 by increasing deliberateness of thedriving behavior of autonomous vehicle 10. The method may then return toblock 222 to continue detecting the first property of passenger 14.

FIG. 26 shows an exemplary embodiment of a method 2400 for modifyingdriving behavior of an autonomous vehicle based on passengersatisfaction of passenger 14. In block 220, a system 50 is providedcomprising first sensor 16, processor 18, and automated driving system52. In block 222, first sensor 16 is used to detect a first property ofpassenger 14. In block 224, processor 18 calculates passengersatisfaction index PSI based on the first property detected by firstsensor 16. In block 236, it is determined whether the passengersatisfaction index PSI meets a predetermined level of passengerdissatisfaction. If the passenger satisfaction index PSI does meet apredetermined level of passenger dissatisfaction (“Yes” in block 226),then the method proceeds to block 240. If the passenger satisfactionindex PSI does not meet the predetermined level of passengerdissatisfaction (“No” in block 226), then the method returns to block222 to continue detecting the first property of passenger 14. In block240, processor 18 controls automated driving system 52 to modify drivingbehavior of autonomous vehicle 10 by decreasing a magnitude ofacceleration and/or deceleration of autonomous vehicle 10. The methodmay then return to block 222 to continue detecting the first property ofpassenger 14.

FIG. 27 shows an exemplary embodiment of a method 2500 for modifyingdriving behavior of an autonomous vehicle based on passengersatisfaction of passenger 14. In block 220, a system 50 is providedcomprising first sensor 16, processor 18, and automated driving system52. In block 222, first sensor 16 is used to detect a first property ofpassenger 14. In block 224, processor 18 calculates passengersatisfaction index PSI based on the first property detected by firstsensor 16. In block 236, it is determined whether the passengersatisfaction index PSI meets a predetermined level of passengerdissatisfaction. If the passenger satisfaction index PSI does meet apredetermined level of passenger dissatisfaction (“Yes” in block 226),then the method proceeds to block 242. If the passenger satisfactionindex PSI does not meet the predetermined level of passengerdissatisfaction (“No” in block 226), then the method returns to block222 to continue detecting the first property of passenger 14. In block242, processor 18 controls automated driving system 52 to modify drivingbehavior of autonomous vehicle 10 by increasing a distance betweenautonomous vehicle 10 and nearby or proximate objects. The method maythen return to block 222 to continue detecting the first property ofpassenger 14.

FIG. 28 shows an exemplary embodiment of a method 2600 for modifyingdriving behavior of an autonomous vehicle based on passengersatisfaction of passenger 14. In block 220, a system 50 is providedcomprising first sensor 16, processor 18, and automated driving system52. In block 222, first sensor 16 is used to detect a first property ofpassenger 14. In block 224, processor 18 calculates passengersatisfaction index PSI based on the first property detected by firstsensor 16. In block 236, it is determined whether the passengersatisfaction index PSI meets a predetermined level of passengerdissatisfaction. If the passenger satisfaction index PSI does meet apredetermined level of passenger dissatisfaction (“Yes” in block 226),then the method proceeds to block 244. If the passenger satisfactionindex PSI does not meet the predetermined level of passengerdissatisfaction (“No” in block 226), then the method returns to block222 to continue detecting the first property of passenger 14. In block244, a distance to a nearby object is determined. In block 246, it isdetermined whether the distance to the nearby object is less than apredetermined threshold. If the distance to the nearby object is lessthan the predetermined threshold (“Yes” in block 246), then the methodproceeds to block 248. If the distance to the nearby object is not lessthan the predetermined threshold (“No” in block 246), then the methodreturns to block 244 to continue detecting distance to nearby objects.Alternatively, in another exemplary embodiment, the method may return toblock 222 to continue detecting the first property of passenger 14 todetermine whether further adjustments to driving behavior are necessary(shown as dashed line in FIG. 28). In block 248, processor 18 controlsautomated driving system 52 to modify driving behavior of autonomousvehicle 10 by decreasing a speed of autonomous vehicle 10. The methodmay then return to block 244 to continue detecting a distance to nearbyobjects. Alternatively, in another exemplary embodiment, the method mayreturn to block 222 to continue detecting the first property ofpassenger 14 to determine whether further adjustments to drivingbehavior are necessary (shown as dashed line in FIG. 28).

In accordance with an exemplary embodiment, FIG. 29 shows an embodimentof a system 60 for increasing satisfaction of a passenger 14 in anautonomous vehicle 10 having an automated driving system 52. System 60may include processor 68, automated driving system 52, first sensor 62,second sensor 64, and display 66. Automated driving system 52 isdescribed in detail herein. Processor 68 is similar to processor 18described above, with the addition that processor 18 is structured tocalculate a vehicle path plan based on input from first sensor 62.Additionally, processor 68 may control display 66 to display a graphicalrepresentation of the driving environment of the vehicle and the vehiclepath plan.

First sensor 62 may be operably connected to automated driving system 52and/or processor 68. First sensor 62 may be structured to detect adriving environment of the vehicle, and, as exemplary embodiments, mayinclude a camera, Radio Detection and Ranging (RADAR) system, LightDetection and Ranging (LIDAR) system, or any combination of thesesystems. First sensor 62 detects the surroundings of autonomous vehicle10 so that processor 68 is aware of nearby objects and can create oradjust a vehicle path plan accordingly. Second sensor 64 will bediscussed in detail below.

FIG. 30 shows an exemplary embodiment of display 66 (see FIG. 29).Display 66 may be an LCD screen, LED screen, or other suitable type ofdisplay screen, and can be mounted in autonomous vehicle 10 to bevisible by passenger 14. Display 66 may also be incorporated intoexisting display modules used in vehicles for controlling environmentalor audio features. Display 66 may also be a touch screen device so thatpassenger 14 can make inputs by touching display 66.

Display 66 may provide a variety of information to passenger 14. Forexample, display 66 may show real-time data related to the operation ofautonomous vehicle 10, such as vehicle speed 70 and acceleration status72. In FIG. 30, acceleration status 72 is shown by an arrow indicating adirection of the acceleration, which the magnitude of acceleration beingshown by an amount of shading in an arrow icon.

Display 66 may further show a stylized icon 74 representing autonomousvehicle 10, and stylized icons 76 to represent a relative position ofnearby objects. In FIG. 30, stylized icon 76 shows a vehicle ahead ofautonomous vehicle 10. It will be understood that other types ofstylized icons could also be used, such as icons for pedestrians,bicyclists, road obstacles, etc. Processor 68 (see FIG. 29) may bestructured to classify nearby objects based on information from firstsensor 62 or from vehicle-to-vehicle or vehicle-to-infrastructurecommunication. Display 66 may be further configured to indicateproximity to nearby structures by various levels of shading or color, asshown by shaded region 84. Display 66 may further illustrate a bufferzone 86 to represent a distance maintained between autonomous vehicle 10and nearby objects by processor 18 and automated driving system 52.

Display 66 my further show icons indicating various features ofinfrastructure information. In FIG. 30, for example, display 66 mayinclude an icon 78 indicating a stoplight ahead and an icon 80indicating the local speed limit. However, it will be understood thatmany types of icons indicating a variety of different traffic andinfrastructure conditions may be used. For example, display 66 maydisplay icons representing a yield zone, merging traffic, constructionzones, upcoming roundabouts, intersections, one-way streets, tolls, stopsigns, school zones, etc.

Display 66 may further show display icons 82. Display icons 82 can bepressed by passenger 14 to configure display 66 or activate otheroptions. Display 66 may further show a LIDAR thumbnail 88 that shows areal-time image of the surroundings as detected by a LIDAR system ofautonomous vehicle 10.

Overall, display 66 is structured to provide a simplified andeasy-to-read graphical representation of the information used byprocessor 18 and automated driving system 52 to control the vehicle.FIG. 31 shows a comparison between an exterior driving environment 92,and the graphical representation 94 shown on display 66 corresponding tothe exterior driving environment 92. Based on this structure, passenger66 can become easily aware of the information being considered and usedby processor 18 and automated driving system 52 to control vehicle 10.

Display 66 may be further structured to provide information regarding anupcoming maneuver based on the vehicle path plan. FIG. 32 shows anexemplary embodiment in which, based on inputs from first sensor 62,processor 68 may determine that a stoplight is ahead, and accordinglyalter the vehicle path plan and control automated driving system 52 toslow autonomous vehicle 10. Display 66 may show a notification 90 sothat passenger 14 is aware that autonomous vehicle 10 will deceleratesoon. Additionally, system 60 (see FIG. 29) may provide an audionotification of the upcoming maneuver through the speakers of autonomousvehicle 10. In this way, passenger 14 receives foreknowledge of upcomingmaneuvers by autonomous vehicle 10, and is not surprised by anunanticipated deceleration. This increases the overall satisfaction andtrust of passenger 14 in the operation of autonomous vehicle 10. Incontrast, without a notification of upcoming maneuvers, passenger 14 maybe surprised by a sudden deceleration, and passenger 14 may feelincreased frustration, anxiety, or distrust in the operation ofautonomous vehicle 10. While FIG. 32 shows a notification 90 of“STOPLIGHT AHEAD SLOWING DOWN,” it will be understood that manydifferent types of notifications are possible. For example, notification90 may include notifications of upcoming acceleration, turns, laneshifts, passing maneuvers, highway merge/exit, yield maneuver or anyother maneuver made by autonomous vehicle 10.

FIG. 33 shows another exemplary embodiment in which system 60 includes acommunication node 96. Communication node 96 may be similar in structureand function to communication node 34 described herein, andcommunication node 96 may be structured to communicate with externaldevice 36 via wireless communications. Exemplary embodiments of externaldevice 36 are described in detail herein. In particular, external device36 may be devices provided in local infrastructure such as signs andstoplights to provide traffic and infrastructure information toprocessor 18, which can be used to display information on display 66such as that shown by icons 78, 80 in FIG. 30.

FIG. 34 shows another exemplary embodiment of system 60 in which firstsensor 62 includes LIDAR system 98. LIDAR system 98 may be used fordetecting an external driving environment of autonomous vehicle 10, andthis information can be used by processor 68 and automated drivingsystem 52 to create and/or modify a vehicle path plan. As shown in FIG.35, display 66 may be configurable to display from LIDAR system 98. Forexample, the top view of display 66 in FIG. 35 shows a standard viewhaving LIDAR thumbnail 88. However, in response to an input by passenger14, display 66 can provide an enlarged LIDAR display 302. This allowspassenger 14 to receive a higher level of detail presented by display66. In other words, display 66 may be configurable to provide varyinglevels of detail based on comfort and interest of passenger 14.

As noted herein, system 60 may further include second sensor 64 (seeFIG. 29, FIG. 33, or FIG. 34). Second sensor 64 may be similar instructure and function to first sensor 16 described in detail hereinwith reference to FIG. 1 through FIG. 28. In other words, second sensor64 may be structured to detect a property of passenger 14 as describedin detail herein. Similarly, processor 68 may use the property ofpassenger 14 detected by second sensor 64 to calculate passengersatisfaction index PSI and change driving behavior of autonomous vehicle10, as described in detail herein. Additionally, if second sensor 64 andprocessor 68 detect high levels of dissatisfaction in passenger 14,display 66 may be used as a tool to help alleviate passengersatisfaction. For example, if the passenger satisfaction index PSIindicates a predetermined level of passenger dissatisfaction, processor68 may control display 66 to display more frequent or more detailednotifications 90 regarding upcoming maneuvers as well as providenotifications of modifications made to driving behavior of autonomousvehicle 10. FIG. 36 shows an exemplary embodiment in which display 66includes a notification 90 informing the passenger that passengerfrustration is detected and therefore a following distance is beingincreased. Thus, system 60 can help to improve satisfaction and trust ofpassenger 14 in autonomous vehicle 10, thereby enhancing the userexperience.

In accordance with an exemplary embodiment, FIG. 37 shows a method 2700for increasing satisfaction of passenger 14 in an autonomous vehicle 10having an automated driving system 52. In block 310, a system 60 (seeFIG. 29) including a first sensor 62, processor 68, and display 66 isprovided. In block 312, first sensor 62 detects a driving environment inthe vicinity of autonomous vehicle 10. In block 314, localinfrastructure information is obtained. As described herein, localinfrastructure information may be obtained from external device 36 viacommunication node 96. In block 316, processor 68 calculates a vehiclepath plan based on the driving environment of autonomous vehicle 10 andthe local infrastructure information. In block 318, processor 68controls display 66 to display a graphical representation of the drivingenvironment and the vehicle path plan. As discussed herein withreference to FIG. 30, the graphical representation displayed in block318 may include real-time data of autonomous vehicle 10 such as speed oracceleration, relative positioning of autonomous vehicle 10 and nearbyobjects, local infrastructure information, or notification of upcomingmaneuvers.

FIG. 38 shows an exemplary embodiment of a method 2800 for increasingsatisfaction of passenger 14. In block 320, a system 60 (see FIG. 29)including a first sensor 62, a second sensor 64, processor 68, anddisplay 66 is provided. In block 322, first sensor 62 detects a drivingenvironment in the vicinity of autonomous vehicle 10. In block 324,local infrastructure information is obtained. In block 326, processor 68calculates a vehicle path plan based on the driving environment ofautonomous vehicle 10 and the local infrastructure information. In block328, processor 68 controls display 66 to display a graphicalrepresentation of the driving environment and the vehicle path plan. Inblock 330, second sensor 64 detects a property of passenger 14, asdescribed in detail above. In block, 332, processor 68 calculates apassenger satisfaction index PSI based on the property of passenger 14.In block 334, it is determined whether passenger satisfaction index PSIsatisfies a first condition. If passenger satisfaction index PSI doessatisfy the first condition (“Yes” in block 334), the method proceeds toblock 336. If the passenger satisfaction index PSI does not satisfy thefirst condition (“No” in block 334), the method returns to block 332 tocontinue detecting the property of passenger 14. In block 336, processor68 controls automated driving system 52 to modify a driving behavior ofautonomous vehicle 10. Exemplary embodiments of modification of thedriving behavior of autonomous vehicle 10 are discussed in detailherein. In block 338, processor 68 controls display 66 to display anotification informing passenger 14 of the change in driving behavior.

The systems and methods described herein may also be used to aid inoptimizing measurement of passenger satisfaction, improvement ofpassenger satisfaction, and modification of driving behavior in responseto passenger satisfaction. In an exemplary embodiment, FIG. 39 shows amethod 2900 for increasing satisfaction of passenger 14. In block 350, aproperty of passenger 14 is detected by second sensor 64 (see FIG. 29).In block 352, processor 68 calculates a passenger satisfaction index PSIbased on the property of passenger 14. In block 354, processor 68modifies driving behavior of autonomous vehicle 10. In block 356, theproperty of passenger 14 is detected again after modification of thedriving behavior of autonomous vehicle 10. In block 358, processor 68calculates a revised passenger satisfaction index PSI based on the newproperty detected in block 356. In block 360, the revised passengersatisfaction index PSI is compared to the original passengersatisfaction index PSI to determine whether the modification to thedriving behavior improved the satisfaction of passenger 14. In block362, communication node 96 uploads data regarding the original propertyof passenger 14, the original passenger satisfaction index PSI, themodifications made to the driving behavior, the new property ofpassenger 14, and the revised passenger satisfaction index PSI to acloud storage such as a computer 42 accessible via the Internet. Thedata uploaded by communication node 96 can analyzed by a recurrentneural network and/or using machine learning algorithms in order todetermine the effectiveness of the system 60 at both evaluatingpassenger satisfaction and the effect of modifications of drivingbehavior on passenger satisfaction. This analysis can be used to improveoperation of the system 60 in future operation.

The exemplary embodiments of systems and methods described above resultin significant advantages over conventional systems and methods. Forexample, exemplary embodiments of the present disclosure allow forreal-time evaluation and improvement of passenger satisfaction, therebyenhancing a user experience with the autonomous vehicle and mitigatingany possible apprehension or lack of trust in autonomous vehicles thatmay be exhibited.

While the above disclosure has been described with reference toexemplary embodiments, it will be understood by those skilled in the artthat various changes may be made and equivalents may be substituted forelements thereof without departing from its scope. In addition, manymodifications may be made to adapt a particular situation or material tothe teachings of the disclosure without departing from the essentialscope thereof. Therefore, it is intended that the present disclosure notbe limited to the particular embodiments disclosed, but will include allembodiments falling within the scope thereof

What is claimed is:
 1. A system for modifying driving behavior of anautonomous vehicle based on passenger satisfaction of a passenger, thesystem comprising: a first sensor structured to detect a first propertyof the passenger; a processor operably coupled to the first sensor, theprocessor structured to calculate a passenger satisfaction index basedon the first property of the passenger; an automated driving systemstructured to operate the autonomous vehicle, the automated drivingsystem being operably coupled to the processor; wherein the processor isstructured to control the automated driving system to modify drivingbehavior of the autonomous vehicle in response to the passengersatisfaction index satisfying a first condition.
 2. The system of claim1, wherein the first condition comprises a predetermined level ofpassenger dissatisfaction; and the processor is structured to, inresponse to the passenger satisfaction index satisfying thepredetermined level of passenger dissatisfaction, control the automateddriving system to reduce a magnitude of acceleration of the autonomousvehicle, control the automated driving system to increase a distancebetween the autonomous vehicle and proximate objects, or control theautomated driving system to decrease a speed of the autonomous vehiclein response to a distance between the autonomous vehicle and an externalobject being less than a predetermined threshold.
 3. The system of claim1, wherein the first condition comprises a predetermined level ofpassenger dissatisfaction; and the processor is structured to, inresponse to the passenger satisfaction index satisfying thepredetermined level of passenger dissatisfaction, control the automateddriving system to increase deliberateness of the driving behavior of theautonomous vehicle.
 4. The system of claim 1, wherein the first propertycomprises a passenger frustration index I_(F) or a passenger trust indexI_(T).
 5. The system of claim 4, wherein the passenger trust index I_(T)is a function of a road monitoring duration value, a secondary activityduration value, a multi-task activity transaction value, a side windowglance value, or a facial gesture value.
 6. The system of claim 5,wherein the passenger trust index I_(T) is given by:I _(T) =W ₃(DMR)+W ₄(DSA)+W ₅(MAT)+W ₆(GSW)+W ₇(FGV) wherein DMR is theroad monitoring duration value, DSA is the secondary activity durationvalue, MAT is the multi-task activity transaction value, GSW is the sidewindow glance value, FGV is the facial gesture value, and W₃, W₄, W₅,W₆, and W₇ are weighting functions for scaling and normalization.
 7. Thesystem of claim 6, wherein the first sensor is a camera; and the roadmonitoring duration value DMR is given by:${{DMR} = {\Sigma_{i = 0}^{n}\frac{{{EGR}\left( {x(t)} \right)}_{i}\Delta \; t}{t}}};$wherein x(t) is a video time series output by the camera, EGR( ) is afunction that outputs a first value if the passenger is glancing at theroad, Δt is a duration of the eye glance to the road, and t is aduration of the video time series.
 8. The system of claim 6, wherein thefirst sensor is a camera; and the secondary activity duration value DSAis given by:${DSA} = {\sum_{i = 0}^{n}\frac{EGP{D\left( {x(t)} \right)}_{i}\Delta t}{t}}$wherein x(t) is a video time series output by the camera, EGPD( ) is afunction that outputs a first value if the passenger is performing asecondary activity, Δt is a duration of performing the secondaryactivity, and t is a duration of the video time series.
 9. The system ofclaim 6, wherein the first sensor is a camera; and the multi-taskactivity transaction value MAT is given by:MAT=Σ _(i=0) ^(n)(EGR(x(t))_(i) +EGPD(x(t))_(i) +EGVD(x(t))_(i)+EGSW(x(t))_(i)) wherein x(t) is a video time series output by thecamera, EGR( ) is a function that outputs a first value if the passengeris glancing at the road, EGPD( ) is a function that outputs a secondvalue if the passenger is performing a secondary activity; EGVD( ) is afunction that outputs a third value if the passenger is glancing at avehicular device; and EGSW( ) is a function that outputs a fourth valueif the passenger is glancing to side windows.
 10. The system of claim 6,wherein the first sensor is a camera; and the side window glance valueGSW is given by:GSW=Σ _(i=0) ^(n)(EGSW(x(t))_(i)) wherein x(t) is a video time seriesoutput by the camera, and EGSW( ) is a function that outputs a firstvalue if the passenger is glancing to side windows.
 11. The system ofclaim 6, wherein the first sensor is a camera; and the facial gesturevalue FGV is given by:FGV=Σ _(i=0) ^(n) V(FAC(x(t))_(i)) wherein x(t) is a video time seriesoutput by the camera, FAC( ) is a facial expression of the passenger;and V( ) is a function that outputs a first value if the facialexpression is one of a first group of facial expressions, and outputs asecond value if the facial expression is one of a second group of facialexpressions.
 12. The system of claim 4, wherein the passengerfrustration index I_(F) is a function of a galvanic skin response value,a skin temperature value, a verbal valence value, or a facial gesturevalue.
 13. The system of claim 12, wherein the passenger frustrationindex I_(F) is given by:I _(F) =W ₈(GSR)+W ₉(ST)W ₁₀(VV)+W ₁₁(FGV) wherein GSR is the galvanicskin response value, ST is the skin temperature value, VV is the verbalvalence value, FGV is the facial gesture value, and W₈, W₉, W₁₀, W₁₁ areweighting functions for scaling and normalization.
 14. The system ofclaim 13, wherein the first sensor is a skin response sensor structuredto output a signal indicating a galvanic condition of the passenger'sskin; and the galvanic skin response GSR is given by:GSR=Σ _(i=0) ^(n) F(x(t))_(i) wherein x(t) is a signal time series ofthe signal output by the galvanic skin sensor; and F( ) is a functionthat outputs a first value if a signal level satisfies a firstpredetermined criteria.
 15. The system of claim 13, wherein the firstsensor is a temperature sensor structured to output a signal thatindicates a temperature of the passenger's skin; and the skintemperature value ST is given by:ST=Σ _(i=0) ^(n) F(x(t))_(i) wherein x(t) is a signal time series of thesignal output by the temperature sensor; and F( ) is a function thatoutputs a first value if a signal level satisfies a first predeterminedcriteria.
 16. The system of claim 13, wherein the first sensor is amicrophone; and the verbal valence value is given by:VV=Σ _(i=0) ^(n) S(Verbal(x(t)))_(i) wherein x(t) is a sound time seriesof the output of the microphone; Verbal( ) is a word spoken by thepassenger; and S( ) is a function that outputs a first value if the wordspoken by the passenger is one of a first group of words, and outputs asecond value if the word spoken by the passenger is one of a secondgroup of words.
 17. The system of claim 13, wherein the first sensor isa camera; and the facial gesture value FGV is given by:FGV=Σ _(i=0) ^(n) V(FAC(x(t))_(i)) wherein x(t) is a video time seriesoutput by the camera, FAC( ) is a facial expression of the passenger;and V( ) is a function that outputs a first value if the facialexpression is one of a first group of facial expressions, and outputs asecond value if the facial expression is one of a second group of facialexpressions.
 18. The system of claim 1, further comprising acommunication node structured to communicate with an external device;wherein the communication node is operably coupled to the processor;wherein the first sensor is a smart device worn by the passenger;wherein the communication node is structured to communicate with thesmart device to receive the first property; and wherein the firstproperty comprises a passenger galvanic skin response, a passenger skintemperature, or a passenger heart rate.
 19. The system of claim 1,further comprising a communication node structured to communicate withan external device; wherein the communication node is operably coupledto the processor; wherein the communication node is structured toreceive traffic data, weather data, passenger social data, passengercalendar data, or destination data from the external device; and whereinthe processor is structured to modify the passenger satisfaction indexbased on the traffic data, the weather data, the passenger social data,the passenger calendar data, or the destination data.
 20. A method formodifying driving behavior of an autonomous vehicle based on passengersatisfaction of a passenger, the method comprising: providing anautonomous vehicle comprising a first sensor, a processor, and anautomated driving system structured to operate the autonomous vehicle,the first sensor, the processor and the automated driving system beingoperably coupled; detecting, with a first sensor, a first property ofthe passenger; calculating, with a processor, a passenger satisfactionindex based on the first property of the passenger; controlling theautomated driving system to modify driving behavior of the autonomousvehicle in response to the passenger satisfaction index satisfying afirst condition.